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What should it cost to send a bit of data across the Internet? It 


sounds like a simple question, 


Networks involve a mix of fixed and variable investments, 
requires assumptions about demand levels, 
The entire Internet industry depends on communications networks, 


but the answer is surprisingly complex. 


and pricing 
competition and usage patterns. 
but the 


economics of those networks are poorly understood and likely to change. 


High-level trends are obvious: 
prices. 


The most interesting phenomena, however, occur locally. 


Convergence, growing capacity and falling 


Despite 


the abundance of bandwidth, most users still experience congestion. 


A number of companies are working to make the Internet more scalable and 


more reliable. 


Through their effects, 
easily bought and sold in multiple configurations. 


bandwidth will become a commodity, 
Companies will be 


able to prioritize traffic to ensure that critical information gets 


through, 


intolerant services such as voice and streaming video. 


and the Internet will increasingly become a vehicle for delay- 


Simplicity at the 


user level, however, will mask growing complexity behind the scenes. 


The communications industry has 
evolved steadily for such a long 
time that it is hard to imagine 
discontinuous change. Yet rapid 
transformation is a very real pos- 
sibility in the next decade. The 
outcome will determine whether the 
Internet can grow into a true mass 
medium and a universal platform for 
commercial activity. 


The physical, legal and financial 
underpinnings of the traditional 
voice network are collapsing (see 
Appendix, page 21). The cost-plus 
pricing models of the telecommuni- 
cations cartels are bound to fall 
amid the dual onslaught of technol- 
ogy and deregulation. Data, not 
voice, will be the primary form of 
traffic, and many suppliers will 
compete to offer ever-faster con- 
nections. Band-X and RateXchange, 
who have created markets to trade 
spare capacity, provide a glimpse 
of this future. 
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But the story doesn’t end there. The Internet is a complex adaptive sys- 
tem, comprising many interconnected and interdependent parts. To the maxi- 
mum extent possible, traffic seeks routes where there is capacity, shifting 
around the globe as night moves west. But without technical and financial 
structures to facilitate more efficient arrangements, there is always the 
possibility of overload. To users a network is only as good as its slowest 
link, and the Internet has many slow links. Congestion will remain a prob- 
lem for the foreseeable future, but this will also create opportunities for 
companies such as IPHighway, Orchestream, CLASS Data Systems, InterNAP and 
SkyCache that provide mechanisms to get around bandwidth bottlenecks. 


Prices overall will drop (no surprise there), but they will also evolve and 
differentiate in unpredictable ways. Declining wholesale costs almost never 
translate directly to retail price reductions: The price of a call from 
the UK to the US has fallen 70 percent over the past decade, but trans- 
Atlantic cable costs on a per-voice channel basis have gone from $9000 to 
$125. What you charge depends heavily on who you are, and where your prof- 
it margins come from. 


In an earlier issue, we predicted that bandwidth would generally be abun- 
dant, but that there would be frequent periods of scarcity (see Release 1.0, 
12-95). Indeed, since that time capacity has skyrocketed, but so has 
demand. This trend shows no sign of abating. 


More people will use the Internet two or five or 20 years from now, and 
those users will also require more bandwidth and devices. First, as higher- 
bandwidth network-access technologies such as digital subscriber line (DSL) 
and cable modems become available to the residential mass market, individual 
users will place greater demands on the network. Ten thousand users with 
one-megabit-per-second (mbps) connections could require 10 gigabits per sec- 
ond (gbps) of total capacity if everyone were online simultaneously, far 
more than any Internet backbone provides today. The exponential scaling of 
demand poses a far greater challenge than the arithmetic growth of users. 
Second, applications such as streaming video will become viable for the mass 
market, further bumping up capacity requirements. Finally, as networks 
evolve from dial-up connections to “always-on” data-oriented environments, 
applications will increasingly use capacity even when users are not present. 
Computer-to-computer traffic will drive demand, with more and more devices 
becoming Internet-enabled. As UUNET chief scientist Mike O’Dell points out, 
“The fundamental error people make is believing that humans have something 
to do with this.” 


The year of convergence: 1984-2002 


Analysts have proclaimed the imminent arrival of convergence for two 
decades. Everyone agrees that separate voice, video and data networks will 
soon be replaced by... something. Yet the definitions of convergence have 
somehow never seemed to converge. The telephone network engineers thought 
they would lead the way to nirvana with the Integrated Services Digital 
Network (ISDN), or Asynchronous Transfer Mode (ATM) switching, or perhaps 
video dialtone. The cable companies rhapsodized about “full-service net- 
works” using their broadband connections to enable the killer apps of home 
banking and video-on-demand. Internet experts knew that push media and 
WebTV would soon sweep the world. Governments figured that removing legal 
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barriers would generate rampant cross-media competition, with multiple 
broadband wires sprouting affordably into every home. 


It hasn’t happened that way. At least, not yet. Convergence is real, but 
it’s not nearly as simple as everyone thought. The Internet today still 
depends heavily on the economics and regulatory dynamics of the old voice 
telephone network (see Appendix, page 21). The nagging problem is that net- 
work operators desire a return on their investments. The necessary infra- 
structure costs real money, and the return on investment has always been 
somewhat speculative. Companies with lucrative franchises have little 
incentive to cannibalize their own revenue streams absent a serious competi- 
tive threat. The result is a sort of prisoner’s dilemma. None of the com- 
panies best able to make convergence happen feels the need to act absent a 
competitive threat from one of the others. 


Con-vergence » not mono-vergence 


The crucial point is that convergence means heterogeneity rather than homo- 
geneity. True, one network can now carry multiple kinds of traffic. Such 
networks, however, will often be more complicated than networks optimized 
for a single function. Each kind of traffic has particular usage patterns, 
cost elasticities and alternative media. Broadband data networks are not 
deployed in a vacuum; they must interconnect with legacy infrastructure and 
regulatory environments. Service providers will have a harder time guaran- 
teeing quality of service when traffic must traverse multiple networks, 
using different technologies and equipment from many different suppliers 
(see page 6). The Internet’s distributed, layered architecture facilitates 
rapid innovation at the edge of the network, but also leads to greater 
intricacy. This complexity will only increase as peering, which let traffic 
shift around easily, is replaced by settlements that provide economic incen- 
tives for allocating idle bandwidth (see page 10). 


Change in complex systems tends to be discontinuous, in what biologists call 
“punctuated equilibrium.” An environment may appear stable for long periods 
of time, but shift rapidly once a tipping point is reached. The tradition- 
al telecommunications industry, which generated $220 billion in service rev- 
enue for US carriers and $739 billion worldwide in 1996, is entering a 
period of profound uncertainty. Data bandwidth will exceed voice by the end 
of this year, and will represent over 90 percent of global traffic in less 
than a decade, according to researchers Philip Mutooni and David 
Tennenhouse. Over many routes data already predominates; voice capacity 
between Sweden and the US, for example, is approximately 50 mbps, but 
Internet capacity exceeds 800 mbps. 


Despite all this, voice still represents the vast majority of telecommunica- 
tions revenue. The over-$100 billion US long-distance market alone dwarfs 
the $9 billion in Internet access revenues in 1997. Voice prices today are 
arbitrary because of the influence of regulators and the absence of competi- 
tion (see Appendix, pages 22-25). Per-minute pricing for local calls is 
anathema for most Americans, yet it is the standard in most of the world 
over the same kinds of networks. Even within the US, prices vary wildly 
from state to state, independent of geographic cost differences. As unregu- 
lated data services represent a greater and greater share of usage, however, 
something is bound to shift. 
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SCARCITY OR ABUNDANCE? 


Several technical breakthroughs including wave-division multiplexing (WDM), 
which multiplies available capacity over fiber-optic networks using multiple 
frequencies of light, have greatly reduced transmission costs. “New” telcos 
such as Qwest, Level 3 and Williams Communications have all announced plans 
to build massive fiber backbones across the US, and several companies are 
deploying large networks across Europe and Asia. Submarine cables including 
Project Oxygen (see page 5) promise to multiply international capacity many- 
fold. Low-earth-orbiting satellite systems such as Teledesic (see Release 
1.0, 2-95) plan to offer global high-speed connections by 2003. To support 
these networks, a new crop of start-ups such as Pluris, Avici and Juniper 
plan to build routers capable of handling over a terabit per second (tbps). 
Industry observer David Isenberg, who recently went independent after 12 
years at AT&T Bell Labs, says that the combination of new capacity and new 
entrants uncommitted to the status quo means that “all the ingredients of an 
unstable market appear to be there.” 


It’s all about tradeoffs 


Economics is the study of choice under scarcity. Without scarcity, individ- 
uals need not weigh costs and make tradeoffs. Whenever capacity is cheap, 
providers will simply throw bandwidth at their problems. Qwest can provide 
high-quality Internet telephony because its massive and so-far uncrowded 
network can transmit packetized voice in uncompressed 64 kilobits-per-second 
(kbps) streams without breaking a sweat. When bandwidth is scarce, compa- 
nies will develop clever techniques to save it, just as hard-disk compres- 
sion software became wildly popular for a short period when drive capacity 
fell behind software size. 


Drilling deeper, communications prices reflect the balance of transmission, 
switching and storage costs. Each factor can to some extent substitute for 
the others if it is cheap enough on a relative basis. With enough band- 
width, it wouldn’t even make sense to route each user’s data to separate 
endpoints. Rather, much like radio, everyone’s information would be broad- 
cast throughout the network, and users would “tune” into their own encrypted 
datastream. (This is the architecture of “passive optical networks” that 
use WDM over fiber connected directly to customers’ homes.) 


The falling price of bandwidth will shift the economics of the Internet well 
before that point. From a business standpoint, abundant bandwidth gives 
power to those vendors, mostly on the edge of the network, who offer value- 
added services. What Isenberg calls the “stupid network” arose because in 
an inter-networked environment many cheap computers can do a better job than 
the huge central switches of the telephone network. Carriers will no longer 
be able to control pricing and application functionality simply because they 
are responsible for moving bits. 


With skyrocketing demand, however, the balance has shifted. Following 
Moore’s law, semiconductor performance has indeed doubled every 18 months, 
but bandwidth requirements on Internet backbones have doubled every four 
months. As a result, service providers have added more processing and com- 
plexity to the network to manage capacity better. Caching, private inter- 
connection points and policy-based routing (see pages 7-13) are all at- 
tempts to save scarce bandwidth. The amount of money pouring into new 
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backbone networks suggests that many investors believe a bandwidth glut is 
unlikely. But the current equilibrium is unstable. Technological or com- 
petitive developments in one area will change the dynamics, creating new 
market opportunities until the system returns to temporary balance. 


Project Oxygen 


Seventy percent of the Earth’s surface is water, and sending bits across 
oceans is difficult. Satellites suffer from relatively limited capacity 
and high latency (response time). Submarine cables are the other alter- 
native. Carriers have invested some $15 billion dollars laying ever- 
bigger cables over the past decade. Because each cable costs hundreds 
of millions of dollars and takes years to build, cartels have tradition- 
ally split the cost and pre-allocated capacity through 25-year leases. 


New Jersey-based CTR Group has audacious plans to change all that. 
Project Oxygen will be a global “super-Internet” of submarine cables 
around the globe, with major segments operational by the year 2000. By 
2002, CTR Group expects to spend $8 billion to deploy 158,000 kilometers 
of cable connecting 101 landing points in 74 countries. Founder Neil 
Tagare wrote the feasibility study for the $2 billion Fiber Optic Link 
Around the Globe (FLAG), which greatly increased submarine cable capaci- 
ty between Europe and Asia. FLAG was the first privately-funded cable 
system; Project Oxygen pushes the model much further. 


Rather than doling out point-to-point capacity, Project Oxygen will make 
international bandwidth available to carriers and companies on demand. 

A customer will simply purchase a given level of capacity and shift it 
to any segment of the network it desires. 


According to Tagare, “The one thing every phone company has in common is 
that they are always wrong in their forecasts.” Some 50 percent of car- 
rier investments in international capacity are unnecessary, because 
demand patterns do not follow long-term projections. The growth of the 
Internet has worsened this problem, because Internet traffic grows con- 
siderably faster and more unpredictably than voice. In effect, Project 
Oxygen’s network model will average out the risk of misjudging capacity 
needs on any route, creating a more predictable market for carriers. If 
a particular link on Project Oxygen does become congested, traffic can 
be sent over alternate routes until more capacity is added. 


The most impressive aspect of Project Oxygen is the sheer volume of 
bandwidth it will provide. Minimum capacity will be 160 gbps on each 
fiber pair and 640 gbps on each cable segment. By comparison, total 
trans-Atlantic capacity in December 1995 was 36 gbps, and trans-Pacific 
capacity was 9 gbps for submarine cables and satellites combined. 
Project Oxygen’s pricing will also dramatically undercut current rates. 
CTR Group is taking orders for capacity now, with prices on a sliding 
scale based on volume, and expects to close its first round of financing 
commitments by December. 
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SCALING AND QUALITY OF SERVICE 


The fundamental challenge for the Internet is scaling. Both demand and sup- 
ply will continue to grow, and approaches that once worked may break down 

as the numbers get bigger. Companies must understand networks holistically, 
and they must find ways around bottlenecks that do not become bottlenecks 
themselves. In a heterogeneous environment there can be no magical acceler- 
ator pedal. There are, however, a number of second-best solutions to make 
more efficient use of the bandwidth we have. 


Not all bits are created equal. You might not mind if your e-mail message 
arrives a minute late, but if one second of your real-time voice conversa- 
tion gets delayed you may never get it back. Similarly, a business that 
has invested large sums into its electronic-commerce Website will be more 
concerned about reliable connections than a residential user with a personal 
home page. Quality of service (QOS) management seeks to match network per- 
formance with a particular user’s expectations, at a price. 


The Internet was designed as a “best-effort” network. There is no guarantee 
that any given packet will reach its destination; packets not arriving are 
simply re-sent. This system avoids the administrative overhead of the pub- 
lic switched telephone network, but also limits the utility of the Internet 
for mission-critical applications. Even when companies are willing to pay 
more for enhanced QOS, they often cannot get it today. The greatest barri- 
er is that any single Internet transmission usually passes through multiple 
networks. Even if the originating service provider deploys sufficient band- 
width and can guarantee robust QOS, the user gains no benefits if the other 
providers in the chain don’t do so too. QOS is easier to manage inside 
private networks, but even there information usually must traverse multiple 
devices and wide-area network (WAN) links. 


On the road to a more reliable Internet 


A small category of users are big enough to create QOS on their own. 
For example, the major players in the US automotive industry plan to 
launch a massive private network later this year to provide high-quality 
connections to their thousands of trading partners. The industry 
believes that, by developing a single network standardized on the 
Internet protocol (IP), it can dramatically reduce costs compared to 
current proprietary electronic data interchange connections. The 
Automotive Network Exchange is being developed in conjunction with 
Bellcore, which will require participating Internet service providers 
(ISPs) and exchange points to meet certification requirements in eight 
categories: Network service features, interoperability, performance, 
reliability, disaster recovery, security, customer care and trouble han- 
dling. 


QOS doesn’t eliminate the need for more bandwidth; it simply makes it possi- 
ble for some users to get more reliable service. Most Internet traffic 
will still use “coach class,” so ISPs will have to provision sufficient 
capacity for the baseline service to remain acceptable. 


A few years ago, ATM and the Internet Engineering Task Force’s (IETF’s) re- 
source reservation setup protocol (RSVP) were expected to solve QOS con- 
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cerns. Both have their place, but neither has proven to be a magic bullet. 
ATM, which emerged from the telephone network world, is a switching protocol 
with sophisticated QOS management features. But this functionality comes at 
the price of significant complexity and overhead. Most companies have cho- 
sen Ethernet instead of ATM for their local-area networks, and are unwilling 
to replace their existing hardware. RSVP allows applications to request 
that sufficient bandwidth be reserved for particular traffic flows. To take 
advantage of RSVP, however, all routers and applications in a network must 
be upgraded. Both ATM and RSVP also manage QOS on a per-flow basis, a 
technique that doesn’t scale well enough for the core of the Internet. As 

a result, work in the IETF has shifted toward a simplified approach known 

as Differentiated Services (Diff Serv). 


More than a dozen companies are developing QOS solutions of one form or 
another, and all the major router vendors have incorporated QOS features 
into their products. The biggest problem has been supporting QOS on an 
end-to-end basis in today’s networks. Few network managers have taken 
advantage of the QOS features in routers and switches because of the com- 
plexity of configuring multiple devices, and getting their counterparts to 
do so. What is needed is a business-oriented abstraction layer that hides 
the complexity of the process. 


New products forthcoming from start-ups IPHighway, Orchestream and CLASS 
Data Systems (acquired by Cisco in early May) promise to achieve this goal. 
All three propose software-based management systems to configure QOS across 
entire networks. With any of the three products, a network administrator 
can use a simple graphical user interface to set policies for specific 
users, applications, times, etc. For example, an investment bank might want 
urgent trades to receive higher priority across a WAN than packets associat- 
ed with large streaming videos. 


All three technologies work both within corporate networks and across WANs 
operated by ISPs. From a business-model perspective, however, Orchestream 
is focused primarily on selling licenses to service providers and reaching 
companies on a resale basis, while the other two companies intend to market 
to both kinds of customers. Both IPHighway and Orchestream also plan to 
offer inter-provider modules that will allow ISPs to exchange QOS requests 
and billing information, paving the way for more sophisticated settlement 
agreements between providers (see pages 10-12). 


All three companies profess to be agnostic as to QOS technology, rather than 
requiring networks to be upgraded to support RSVP or some other standard. 
The challenge for all of them is create solutions that work in existing 
heterogeneous networks, even though open standards are not yet available for 
all the necessary functions. The companies are working with the IETF on 
standards that will enhance the functionality of their offerings, but in the 
interim they offer proprietary solutions. 


Orchestream and CLASS Data Systems both hope to leverage the dominance of 
Cisco routers in IP networks, the latter by becoming part of Cisco. Many 
Cisco interfaces are open, however, and these systems must all support 
devices from multiple vendors, so CLASS Data Systems’ new identity by no 
means locks out the competing products. IP Highway is more ambitious about 
inserting its policy servers as an addition to existing infrastructure, 
although it claims to be completely transparent to the network. 
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As the three companies bring products to market, their approaches seem to be 
converging. Real-world customers will provide the acid test, although the 
potential market is big enough to accommodate multiple players. 


IPHighway 


Shai Herzog began working on Internet quality of service issues four years 
ago while a graduate student at the University of Southern California, and 
later was the co-author and driving force behind RSVP. Last summer he and 
David Zvilichovsky co-founded IPHighway to develop end-to-end QOS management 
systems. IPHighway is backed by the BRM Group, which funded Check Point 
and BackWeb; former Bay Networks ceo Andy Ludwick is also an investor and 
board member. The 25-person company, with an engineering team in Jerusalem 
and corporate offices in the US, plans to release its first products in the 
fourth quarter of 1998. It is currently in testing with a small number of 
customers. Pricing will depend on network size and components purchased, 
with a basic package costing around $27,000. 


Herzog acknowledges that RSVP works well for some applications but doesn’t 
scale well across large WANs. He says IPHighway will support RSVP and 
other QOS technologies such as IP Precedence and Diff Serv equally. 
IPHighway’s software is designed make end-to-end QOS easy to manage on any 
network, including corporate intranets, VPNs and large ISPs. 


Using IPHighway’s three software modules, managers can define network-wide 
QOS policies without having to configure individual routers. The Management 
Server works with directories to maintain a central database of QOS informa- 
tion, and provides a graphical user interface for managers to define policy 
rules. The Management Server can also detect traffic subject to QOS agree- 
ments, and issue RSVP requests for those applications that do not support 
RSVP. Policy servers, each of which can control several devices, authorize 
and arbitrate between QOS requests based on those rules. Finally, QOS 
Generators monitor traffic at the edge of the network and can automatically 
create QOS requests as needed. 


IPHighway is pushing for adoption of the IETF’s Common Open Policy Service 
(COPS), which Herzog helped design. COPS will provide a standard mechanism 
for individual switches and routers to outsource their policy decisions to 
policy servers. Cisco and 3Com have announced support, but the protocol 
isn’t expected to be ratified until the end of this year. In the absence 
of COPS, the policy servers must issue bandwidth reservation requests with- 
out knowing the load on each router. 


IPHighway’s technology and other QOS tools will make it possible to differ- 
entiate prices among customer segments, and will make it easier to deploy 
services such as Internet telephony that require low latency. 


Orchestream 


Orchestream’s provisioning system automatically configures the QOS features 
in existing Cisco routers. (Disclosure: Esther Dyson is an investor in 
Orchestream.) The 30-employee UK-based company was founded in February 
1996, although its development team was not in place until April 1997. Or- 
chestream’s system is now in field trials and will be launched commercially 
to a select number of customers at the end of June. CEO Charles Muirhead 
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claims Orchestream/Provider is much less heavily tied to RSVP than other 
approaches, because it can translate QOS requests into existing router man- 
agement interfaces such as Cisco’s Command Line Interface. This, he 
believes, gives the company an advantage in real-world networks, few of 
which support RSVP today. 


Orchestream/Provider offers a simple, central interface for network managers 
to assign QOS levels to users. It then automatically configures the routers 
those managers control. The software-based system uses a combination of 
database technology and the lightweight directory access protocol (LDAP) to 
track service policies necessary to allocate bandwidth to different users. 
Managers use a graphical user interface to define policy objects, which are 
then sent out to local policy servers as needed. 


The system can either “push” policies to traditional routers or can respond 
to policy requests from routers enabled with COPS. Next-generation routers 
may incorporate the Orchestream policy agent directly, eliminating the need 
for a trip across the network to assign QOS to a traffic flow. For scala- 
bility, Orchestream is heavily focused on doing per-flow control only at the 
edge of the network, and aggregating QOS requirements towards the core using 
Diff Serv. 


According to Muirhead, “Today the Internet is like the airline industry with 
only economy seats and no ability to adjust the number of flights per 
week!” Orchestream’s system allows companies to save on costly WAN band- 
width, and also makes new applications such as high-quality Internet teleph- 
ony easier to implement. From the service-provider standpoint, Orchestream 
allows ISPs to differentiate themselves in an increasingly-crowded market. 


Orchestream plans to focus on service providers because they already manage 
most WANs, and because Orchestream believes QOS must start at the core of 
the network before moving to the edge. The company is currently in negoti- 
ations with several major ISPs and carriers, and is also in talks with 
major network equipment vendors. 


CLASS Data Systems 


Like Orchestream and IPHighway, CLASS Data Systems enables end-to-end QOS 
through distributed software connected to a central management server. 

(Also similar to IPHighway, company executives are based in the US but the 
engineering team is in Israel). The primary difference is that the 
CLASSifier system includes intelligent QOS software “agents” that interact 
with application servers, in addition to network devices such as routers. 
The agent software can be installed directly on Windows-based servers, or 
can run on proxy servers that interface with mainframes and other platforms. 


The currently-available “Agent-Based Application QOS” classifies traffic at 
the client-server source, either by network-level parameters such as IP 
address or by application-specific parameters such as application name, URL 
and file name. Although users must install additional software, agents 
allow tighter integration into server applications. For example, different 
transaction types on an Oracle database could receive separate classifica- 
tions. CLASS Data Systems vice president of engineering Gilad Zlotkin 
claims that competing approaches can differentiate only between applica- 
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tions, not on a more specific basis. Once agents classify traffic flows, 
they query policy servers for applicable QOS policies, and then request the 
required service levels from network devices. The system supports both RSVP 
and priority bit QOS standards. 


In May, CLASS Data Systems announced a “Policy to Device Control” option 
which will be available in the third quarter. Companies can deploy the new 
product as either an alternative or a supplement to the agent-based solu- 
tion, depending on their specific network configuration. Policy to Device 
Control implements QOS policies directly on network devices, similar to the 
IPHighway and Orchestream approaches. 


On May 4, Cisco agreed to acquire privately-held CLASS Data Systems for $50 
million. The acquisition gives Cisco an end-to-end, easy-to-use QOS solu- 
tion to complement the QOS functionality in its routers. Although unwilling 
to discuss specific plans, Zlotkin says that “Cisco bought us for what we 
have and also for what we plan to have.” As part of Cisco, CLASS Data 
Systems will be able to integrate its software more closely with the fea- 
tures in Cisco devices. 


INTERCONNECTION AND PEERING 


The Internet is, by definition, a network of networks. Some networks are 
operated by ISPs, while others are run by corporations. Some networks pro- 
vide connectivity to end users, while others carry traffic between end-user 
networks. However, just as light behaves as both a particle and a wave, 

the Internet appears to be both flat and hierarchical. What seems to be a 
private end-user access network can also function as a transit network. The 
problem for traffic management is that the Internet has many edges but no 
center. Traffic is routed dynamically, based on a combination of algorithms 
and business relationships among multiple providers. 


Historically, today’s Internet evolved from the NSFNET, a US-government 
funded research and educational network. The NSFNET backbone carried traf- 
fic between regional research networks across the US and elsewhere. Over 
time, corporations involved in research projects began to hook up their net- 
works to the NSFNET, still operating nominally under a non-commercial 
“acceptable use” policy. Eventually commercial IP networks emerged, begin- 
ning with the spinout of PSINet from NYSERNet, the New York regional net- 
work. Finally, the US government phased out its control of the NSFNET, 
completely turning over backbone functions to private entities by 1995. As 
part of the transition, the National Science Foundation funded the creation 
of several public network access points (NAPs), and directed backbone 
providers to exchange traffic for free at those points (an arrangement known 
as peering). Privately-funded multilateral exchange points also emerged 
around the same time, in particular MFS Communications’ metropolitan area 
ethernets (MAEs). 


In the three years since NSFNET privatization, the number of backbone 
providers has increased dramatically. Boardwatch magazine counts over 30 US 
national providers in its survey. The reality, however, is that a handful 
of those providers — UUNET, MCI, Sprint, GTE (BBN), PSINet, and AGIS — con- 
trol the vast majority of routes, and the first three on that list carry 
some three-quarters of Internet traffic. Peering only makes sense for pro- 
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viders of roughly equal size; a small network gets much greater benefit from 
the ability to reach subscribers on a bigger network (see Release 1.0, 12- 
95). There is, however, no bright line between “Tier 1” providers entitled 
to peering and “downstream” ISPs that must pay those providers to transit 
their traffic. The value of peering for two parties depends on more than 
simply their relative numbers of customers, and the administrative costs of 
alternative arrangements must be factored in. 


UUNET generated a great deal of controversy last year when it revoked peer- 
ing agreements with several smaller networks and demanded that they pay for 
their connections as transit customers. UUNET may have gained the most 
notoriety, but all the large backbone providers are increasingly unwilling 
to enter into new peering agreements. Major carriers (but less-prominent 
Internet players) such as GTE and AT&T have found themselves unable to peer 
with the major backbones, until they purchased existing providers (BBN and 
CERFNet, respectively) that had peering arrangements in place. The Tier 1 
backbones have also negotiated bilateral private peering arrangements with 
each other, decreasing the amount of traffic they send through the multilat- 
eral public exchange points. 


What’s really at stake here 


Whether the Tier 1 providers’ refusal to peer constitutes anti-competitive 
activity is an important question. We certainly think companies’ peering 
policies should be out in the open, rather than subject to non-disclosure 
agreements as they usually are today. The backbone market has evolved out- 
side the framework of telecommunications regulation, and so far the FCC has 
stayed out of peering disputes. The proposed merger of MCI and WorldCom 
has, however, brought greater attention to the competitive dynamics of this 
area. WorldCom has in recent years purchased the networks of UUNET, ANS 

and CompuServe, all major backbone players. Its combination with MCI would 
have made it by far the largest backbone (carrying end-to-end traffic for 62 
of the top 100 Internet sites). American and European antitrust regulators 
raised questions about the competitive effects of the deal, and WorldCom 
agreed to sell MCI’s backbone operations to Cable and Wireless for $625 mil- 
lion. As we go to press, however, the European Commission’s top antitrust 
official, Karel van Miert, has said the sale doesn’t go far enough, so 
we’ll see how all this plays out.... 


Beyond the competitive issues, public exchange points are becoming a signif- 
icant performance bottleneck. As usage levels grow, these multilateral con- 
nections are increasingly congested. Moreover, as private peering becomes 
more prevalent, large players find fewer benefits to interconnecting at NAPs 
and MAEs. Companies increasingly engage in “hot potato” routing, handing 
off traffic to other backbones at the first possible opportunity regardless 
of the potential routing inefficiencies. The problem is that there is no 
organized system of interconnection pricing, leading to confusion and inef- 
ficient arrangements. Everyone recognizes the need for more economic and 
efficient traffic exchange mechanisms, but it is hard for any one provider 
to put them into place. 


Enter the peering brokers. Savvis was the first company to build an Inter- 
net backbone out of private connections with Tier 1 providers, completely 
bypassing the NAPs. Savvis turned heads when it topped a Boardwatch back- 
bone performance survey. Mike Gaddis of Savvis has now formed the Brokered 
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Private Peering Group (BPPG) to expand the use of private peering. The 
group’s goal is make private peering more economical for smaller backbones, 
replacing the overburdened NAPs with negotiated connections to private ATM- 
based interconnection points. The BPPG architecture divides bandwidth among 
connected ISPs, but once packets leave the interconnection point they may 
still end up passing through an overburdened public exchange. 


InterNAP 


InterNAP, a 30-person company based in Seattle, believes its architecture 
provides a better answer. According to CEO Anthony Naughtin, the problem 
today is that most major Internet backbones are run by carriers. Original 
backbone operators such as ANS, Netcom, BBN and UUNET have sold out to 
facilities-based carriers in order to gain control over their transport 
costs. However, such facilities-based thinking doesn’t help when traffic 
must be routed across multiple backbones. The performance bottlenecks on 
the Internet today reside at the exchange points. 


InterNAP does not operate any long-haul backbone facilities, nor does it 
request free peering. Instead, it pays the eight largest backbones for con- 
nections into its own private network access points (PNAPs). To these back- 
bones, which collectively represent over 90 percent of Internet routes, 
InterNAP appears as a typical transit customer. In reality, however, virtu- 
ally all the traffic InterNAP sends to a Tier 1 backbone is destined for 
customers on that network. 


InterNAP’s proprietary route management system, Assimilator, works in con- 
junction with routers to optimize traffic flows. Today packets may travel 
across the country or the world to reach a public network access point even 
though their eventual destination is in the same neighborhood. If a packet 
comes into a PNAP from the MCI backbone destined for a customer on AGIS, 
InterNAP will route the traffic directly over its private connection to 
AGIS. A PNAP, however, is a true multilateral exchange point: InterNAP 
advertises all of its routes to parties on both side. The result is great- 
ly improved performance for InterNAP’s ISP and corporate customers. PNAPs 
are now operational in Seattle and New York, with San Francisco, Washington, 
DC, and Los Angeles scheduled to open later this year. 


Naughtin, who spent four years at regional ISP NorthwestNet, says InterNAP 
has so far gotten good cooperation from the upstream providers, because the 
company pays for its connections and “we are destined to become their 
largest customer.” InterNAP effectively provides the technical and finan- 
cial infrastructure for more efficient routing that no one backbone provider 
can offer itself. Naughtin thinks InterNAP’s architecture overcomes some of 
the key barriers to QOS and VPNs over the public Internet. With funding 
from Hambrecht & Quist, Vulcan Ventures and others, the company plans some 
30 PNAPs by the year 2000, a quarter of them outside North America. The 
company has over 60 large customers including Adobe, Amazon.com, Datek 
Online, Go2net, Microsoft and Nordstrom. 


SkyCache: Caching the caches 
Caching is another important component of efficient Internet bandwidth 


utilization. Caches trade scarce bandwidth for more plentiful storage and 
processing, allowing frequently-used documents to be downloaded locally 
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rather than from remote sites. The higher the “hit rate” on the cache, the 
more bandwidth saved. Several companies now offer caching appliances that 
ISPs and companies can deploy in their networks, either on a standalone 
basis or in conjunction with other functions (such as the Bess filtering 
service, see Release 1.0, 5-98). 


Caching solves some of the Internet’s scaling problems, but caches them- 
selves face scaling challenges. As the level of users and traffic increas- 
es, the cache itself can become a bottleneck. It becomes harder and harder 
for the cache to maintain copies of the pages users are likely to request. 
Hierarchical caching provides a partial solution, using multiple caches to 
aggregate user patterns and thereby store a larger percentage of frequently- 
viewed files. But simply plugging multiple caches into each other creates 
its own set of scalability problems. As more data must be passed between 
local and central caches, the system runs into the same Internet transmis- 
sion bottlenecks that created a demand for caching in the first place. 
Newly-public Inktomi, better known as the search engine technology supplier 
for Hotbot and soon Yahoo!, is bringing its clustering technology to bear on 
this challenge, using multiple machines in parallel to improve speed and 
reliability. The real problem, however, involves the transmission links 
rather than the caches themselves. 


SkyCache’s answer is to use satellites to enhance cache functionality. 
Satellites are ideal because they can broadcast large amounts of one-way 
data over large areas to multiple receivers, without using congested terres- 
trial links. SkyCache founder Doug Humphrey built Digex into a major 
regional ISP before selling it to Intermedia Communications. His new 20- 
person company, based in Laurel, Maryland, will sell a turnkey system for 
$2,500 per month to ISPs that includes a satellite receiver and a local 
caching server. The local caches automatically provide statistics to the 
“master cache” at SkyCache headquarters, where patent-pending Reactive 
Caching technology identifies frequently-requested files that are not 
cached. Those files are then broadcast to all the participating caches 
through a continuous 4 mbps satellite datastream. The traditional way to 
increase cache hit rates is to pre-load popular pages, but pre-loading does- 
n’t work when usage patterns shift. Reactive Caching solves that problem. 
During beta testing Apple’s iMac announcement suddenly generated heavy traf- 
fic on the Apple Website, and SkyCache automatically updated caches with 
those pages. 


The SkyCache system will interface with any existing cache compatible with 
the Internet Caching Protocol, an IETF standard for inter-cache communica- 
tion. SkyCache director of marketing Doug Mohney says the company sees 
itself as an enhancement rather than a competitor to caching solutions from 
companies such as Inktomi, CacheFlow and Network Appliance, because it 
enables users to increase their cache hit rates or link up multiple caches 
at the edges of their networks. Another benefit of SkyCache is that its 
architecture fits perfectly with streaming audio and video services, which 
promise to grow greatly in popularity in the near future. SkyCache bypasses 
congested Internet exchange points that limit the performance of streaming 
media, and satellites are perfect for one-way broadcast content. 


SkyCache is currently transitioning from beta testing to commercial deploy- 


ment, and recently closed a $6-million round of venture funding. 
Competition is on the horizon. InterCache, also based in Maryland, plans to 
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launch a similar system based on technology it developed for the military. 
In Europe, SatinNet has been delivering Usenet news to ISPs via satellite 
for two years. The company is working to develop satellite cache pre-load- 
ing in conjunction with Mirror Image, which places centralized caches at 
Internet exchange points (See Release 1.0, 3-98. Disclosure: Esther Dyson 
is an investor in Mirror Image). Cable modem providers are also exploring 
satellite distribution to supplement caches, although these companies are 
likely to self-provision because of their access to cable operators’ satel- 
lite transponder time. 


DNS = Delays in Network Speed? 


Bandwidth is only one factor in Internet delays. A Bellcore study found 
that network transmission represents only 42 percent of the time it 
takes to retrieve a Web page. According to Bellcore’s Christian 
Huitema, the remainder is divided into page preparation at the originat- 
ing server (33 percent), connection to the remote server (12 percent) 
and most surprisingly the domain name system (DNS) query (13 percent). 


The DNS, which maps mnemonic addresses such as edventure.com onto IP 
numbers used for routing, tends to be taken for granted. Yet as the 
study demonstrates, DNS resolution represents a significant Internet per- 
formance bottleneck. Bellcore also found that 4 percent of DNS queries 
simply fail and must be resent (up from 3 percent last year), and that 
DNS queries represented roughly a quarter of all packet traffic on the 
Internet (and 10 percent of the bandwidth usage). 


There is no easy way to replace every ISP’s DNS server with something 
better. Debates about DNS governance and the creation of new generic 
top level domains to complement .com, .net and .org may provide an 
opportunity to enhance the technical underpinnings of the system. Or 
perhaps DNS will simply lose out in the marketplace to Centraal’s Real 
Name service or Netscape’s forthcoming Smart Browser function. 


THE RESURRECTION OF DISTANCE? 


Conventional wisdom holds that the Internet represents the “death of dis- 
tance” (also the title of an excellent book by Frances Cairncross of the 
Economist). The Internet does not require dedicated connections between 
physical points, so in theory it costs almost the same to send data any- 
where. Once carriers deploy fiber-optic networks, the cost and delay to 
transmit information several thousand miles is virtually the same as to send 
it across a neighborhood. According to the Organization for Economic 
Coordination and Development (OECD), the least expensive domestic rate to 
call anywhere in the US (MCI’s five-cents-per-minute charge on Sundays) is 
now only three-tenths of a cent greater than the average peak rate for a 
local call in Europe. 


However, bandwidth supply has not outstripped demand growth, especially at 
choke points such as international circuits. Connectivity is much better 
in North America than the rest of the world. Ironically, London-based 
Band-X (see page 17) found that locating its bandwidth exchange Web server 
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in Canada provided better performance (and much cheaper hosting rates) than 
in England. It still costs money to deploy capacity, and it costs more 
money to build bigger networks. As long as demand growth continues to 
require new infrastructure, distance may be a factor in bandwidth cost. 


Distance will not, however, necessarily correlate perfectly with geography. 
In a recent study, the OECD found that the round trip time on an Internet 
connection from Singapore was 382 milliseconds to North America, 97 mil- 
liseconds to Europe, but 885 milliseconds to other countries in the 
Asia/Pacific region, reflecting differences in infrastructure deployment.! 
In the future, factors such as location and time of day may influence pric- 
ing, but in surprising ways. 


Digital Island 


Internet congestion creates a dilemma for multi-national corporations. 
Digital Island’s answer is to connect directly over leased lines to ISPs 
in different countries. According to CEO Ron Higgins, the company’s 
Internet backbone is designed specifically for global application 
deployment, rather than for access services. It uses ATM switches at 
the core and routers only at the edge of the network, in order to mini- 
mize the number of hops packets must traverse. The company also offers 
value-added services to make it easier for customers to deploy and 
localize global applications. 


Digital Island located its headquarters and its primary data center in 
Hawaii, twice as close to Asia as the continental US but still subject 
to American law and pricing policies. To provide global connectivity 
and “one-stop shopping” for customers, the company has subsequently 
opened centers in London, New York and Santa Clara. Digital Island has 
more than 50 major customers, including Autodesk, National Semiconductor 
and Cisco. 


How high can you go? 


As commercial ISPs emerged in the US in the mid-1990s, they converged around 
a price point of $19.95 per month for unlimited dial-up connections. During 
the last year, many large ISPs have moved away from this pricing structure, 
by raising the monthly fee (AOL), capping monthly usage (IBM, AT&T WorldNet) 
or both (Netcom, Pacific Bell). AT&T reported that some 4 percent of its 
customers were generating half of the total usage time, because of the long 
periods they remained online. So far, AOL and AT&T report no drop in sub- 
scribership as a result of the changes. But just how much are users will- 
ing to pay, especially when new options such as DSL access are made avail- 
able? 


A project at the University of California at Berkeley may help answer this 
question. The Internet Demand Experiment (INDEX), with $1.3 million from 
the National Science Foundation, Cisco and Pacific Bell, is testing user 
responses to different pricing models. The project will recruit 150 stu- 


1 Internet Traffic Exchange: Developments and Policy (OECD 1998), available 
online at www.oecd.org/dsti/sti/it/cm/prod/traffic.htm. 
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dents, faculty and staff to connect to the university Ethernet over ISDN 
connections (DSL when available). Users can adjust connection speeds in 
real-time up to 128 kbps, but prices increase with speed. Using home-grown 
billing software, researchers will test several pricing models, including 
per-minute, per-byte, burst (an occasional maximum speed) and user-selected 
from among several options. 


Berkeley professor Hal Varian points out that whatever pricing model service 
providers choose, applications will evolve around it (for example, using 
automated pre-fetching at night to circumvent congestion resulting from 
flat-rated pricing). To avoid waste, it’s essential to get the prices right 
the first time. But that requires data about user preferences, which up to 
now no one has collected.... Varian says the initial data from 56 subjects 
over the past two months are statistically very clean and show, not surpris- 
ingly, strong correlations between price and usage levels. The question 
researchers hope to answer is what premium users place on greater speed, and 
what forms of usage-based pricing make them most comfortable. 


Sprint: Pricing by the byte 


On June 2, Sprint announced its Integrated On-Demand Network (ION), which 
will be available to businesses later this year and to residential customers 
in late 1999. The announcement was both revolutionary and uninspiring. On 
the revolutionary side, Sprint is the first incumbent facilities-based car- 
rier to announce a major “data-centric” network overhaul. Sprint says it 
has spent $2 billion over the past five years quietly upgrading its network 
to handle data traffic more efficiently. On the uninspiring side, Sprint is 
building ION around ATM, the traditional telco convergence technology, 
rather than engaging in a more revolutionary reorientation of its network. 
For residential customers and businesses outside urban centers, Sprint will 
still have to gain access to customers through local loops controlled by 
local phone companies (see Appendix, page 24). 


The major selling point for Sprint is that it will be able to transport 
voice and data services, and multiple voice lines, over a single wire. 
Sprint claims its architecture will reduce the cost of a long-distance call 
by 70 percent, although Sprint has given no indication that it plans to 
drop its dime-a-minute long-distance pricing to 3 cents. 


An interesting aspect of the Sprint announcement is the pricing model. 
Sprint will provide metering devices to end-users, at a cost of approximate- 
ly $200. These boxes will track the bandwidth that subscribers use, and 
Sprint will bill based on bandwidth usage rather than minutes. The question 
is whether users will appreciate this mechanism. Bandwidth makes more sense 
than minutes for data traffic, because merely being connected does not gen- 
erate any incremental costs for the network. However, users accustomed to 
unlimited pricing may balk at any usage charges. 
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BANDWIDTH TRADING 


Eventually, markets rather than providers will set most communications 
prices. The subversive reality of capitalism is that the right price really 
is no more than what someone will pay. (Karl Marx called this decoupling 

of prices and underlying value commodity fetishism). That’s why a PC that 
costs $2,000 today might be worth $500 a year from now, even though the 
machine itself hasn’t changed. 


Companies in competitive markets search out price points that users will 
accept, and then tailor their products to reach those price points. The 
tradition in non-competitive markets such as telecommunications has been the 
opposite: Regulators figure out what something “costs” and then assign a 
price based on a markup factor reflecting the provider’s “rate of return.” 
Costs certainly influence prices, and network architectures certainly influ- 
ence costs. But the relationships aren’t linear. Companies may take dif- 
ferent levels of risk, or may structure their business models differently: 
Microsoft gave away its browser not because it had found a cheaper way to 
develop software than Netscape. Some markets, such as the retail ISP indus- 
try in the US, converge around a default price, leaving service providers to 
compete on features and brand. Others separate out into multiple price 
points for different classes of customers. 


The key factor is the existence of a competitive market. In such a market, 
any price is presumably reasonable, because companies that set prices too 
high or too low are disciplined by the invisible hand. Without a market, 
all prices are presumably artificial. The retail bandwidth market is still 
not competitive for most end-users. Despite competition among service 
providers, incumbent telephone companies continue to provide the vast major- 
ity of local connections. Over time, competitive and technological develop- 
ments will undermine this control. When the dust settles, though, how will 
prices be set? 


At the wholesale level, two companies have established exchanges for carri- 
ers to trade capacity, and several competitors are in the works. 


Band-X 


Band-X, a London-based company launched in July 1997, was the first to 
establish a functioning bandwidth exchange. The company, founded by Marcus 
de Ferranti and Richard Elliot, now has over 2,400 registered users on its 
Web-based trading system. De Ferranti, who previously worked on telecommu- 
nications deregulation issues for the British government, says that the 
Internet enables a new generation of virtual markets, making information 
instantly available anywhere in the world. Global telecommunications dereg- 
ulation has also helped, creating fragmented markets with many players look- 
ing to buy capacity. 


Band-X’s screens display available bids and offers on particular routes, 
without listing the names of the companies placing them. When a user 
wishes to engage in a transaction, he or she calls Band-X which introduces 
the potential buyer and seller. The two parties negotiate a deal, and if 
they reach agreement they pay Band-X a 0.625 percent commission. De 
Ferranti fully expects Band-X to evolve to a real-time market, and Band-X is 
developing plans to enable immediate clearing of transactions. The comp- 
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any, which still consists of de Ferranti, Elliot, and two part-time staff, 
plans to add another person every two months to keep up with growth. 


Major carriers almost never publish their wholesale rates. Many have joined 
Band-X simply to gain information about price trends, even if they do not 
yet engage in active transactions. The Band-X indices, showing price move- 
ments on the top 20 routes out of London and New York, are now available 
through several publications and financial news services as well as on the 
Band-X Website. Carriers provide their rates to Band-X on confidential 
basis, in exchange for a private summary showing how they compare to the 
norm. Band-X computes averages and then publishes indices that show only 
relative price trends. Elliot comes from a financial background, and Band-X 
consciously designed the indices to facilitate trading in derivatives such 
as options and futures. The spot market isn’t yet sufficiently liquid for 
such secondary markets to emerge, but de Ferranti fully expects them to. 

He points out that derivatives account for 20 times the volume of direct 
spot markets in commodities such as oil. 


De Ferranti believes that in the next few years wholesale telecommunications 
prices will fall “through the floor” as Band-X fosters and benefits from a 
fluid market. In just the last few months, a trans-Atlantic El circuit as 
posted on Band-X with a two-year commitment has fallen from $30,000 to 
$6,000. Interestingly enough, many carriers have recently purchased more 
international capacity than they really need, because they can depreciate 
the costs over the 25-year length of traditional Indefeasible Right of Use 
(IRU) agreements. With telecommunications stocks frequently valued based on 
earnings before interest, taxes, depreciation and amortization (EBITDA), 
this trick provides an artificial boost to the bottom line that often bumps 
up stock prices. On the other hand, the result is that many carriers have 
excess bandwidth, which they are willing to sell at reduced rates through 
mechanisms such as Band-X. 


RateXchange 


Former Sprint and MFS executive Sean Whelan says he founded RateXchange, a 
San Francisco-based competitor to Band-X, because he too saw an oversupply 
of wholesale capacity emerging. The growth in backbone bandwidth, Whelan 
feels, will cause carriers to think differently about how they sell capacity 
on their networks. He thinks carriers will focus on the opportunity cost 

of unused bandwidth, along the lines of the airline industry, rather than 
simply looking to recoup their investments. To get there, however, “some- 
body needs to lead the way,” which is where RateXchange fits in. 


RateXchange, which consists of Whelan and two other people, was launched on 
January 5. Although Band-X was first to market, Whelan sees room for mul- 
tiple exchanges, with the real competition being the traditional 

bilateral methods of buying and selling capacity. RateXchange currently has 
about 500 carrier and corporate registrants, with 10 to 20 more joining each 
day. Some 450 bids and offers have been posted, leading to over 40 trans- 
actions. (Like Band-X, RateXchange simply introduces parties and takes a 
commission if they reach agreement.) 


Offers on RateXchange are divided into four service grades: facilities- 


based carriers selling their own capacity; resale carriers; bypass circuits 
(not terminating on the public network); and Internet telephony. Eventu- 
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ally, Whelan sees more standardized performance benchmarks emerging, using 
the same metrics that carriers employ today to test out new routes. As the 
quality of Internet telephony improves, Whelan believes it will eventually 
comprise the majority of transactions on RateXchange. Whelan agrees with 
Band-X that derivatives markets will emerge to reduce risk and improve liq- 
uidity, but believes such developments are three to five years away. 


What’s missing 


The current generation of bandwidth markets is limited. Band-X and 
RateXchange do not yet provide fully anonymous transactions. One reason for 
this limitation is the absence of sufficient trust. Traders in established 
markets such as the New York Stock Exchange trust that their transactions 
will be executed properly and fairly. Some of this trust comes from the 
“backstop” of government regulation through the Securities and Exchange 
Commission. It will be interesting to see whether such existing financial 
regulators will be necessary or sufficient to generate sufficient confidence 
in bandwidth markets. On the one hand, these markets may be able to devel- 
op without comprehensive regulation, because they serve knowledgeable play- 
ers and the exchanges themselves can take steps to reassure their customers 
(similar to lightly-regulated hedge funds). On the other hand, new regula- 
tory bodies — national, international or otherwise — may develop to police 
the new animals of bandwidth markets. 


Another reason anonymous bandwidth transactions are not yet available is 
that no one is quite sure what bandwidth means. There are several quality 
of service variables involved, as well as differing protocols. For band- 
width to function as a true commodity, there must be standards for how 
bandwidth can be measured and provided. For voice traffic, companies such 
as AT&T have developed technologies to “listen” to a line in order to 
ascertain sound quality on a standard scale. Similarly precise standards 
for data traffic are possible, if the relevant parties can agree on them. 


In the voice world, private and governmental organizations such as Bellcore 
and the ITU have long overseen technical standards to ensure interoper-abil- 
ity. Several efforts are underway to create common frameworks in the data 
world and for Internet telephony. IOPS.ORG, a consortium of major Internet 
service providers, has been working to improve inter-provider communications 
in the event of outages. Internet telephony analyst Jeff Pulver recently 
created a memorandum of understanding process for Internet telephony 
providers to develop a common call detail record for billing purposes. As 
with many Internet issues, these are examples of self-governance through 
novel authority structures that compete with traditional governments for 
legitimacy. 
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VIRTUAL INFRASTRUCTURES 


The death of the independent ISP has been widely proclaimed, but the number 
of ISPs and backbone providers continues to increase. Large national serv- 
ices cannot offer the same level of personal contact and customer support as 
small ISPs, and thanks to rapid growth they often do not provide the same 
level of performance. The result is an inverted bell curve distribution, 
with the big ISPs getting bigger at the same time as small providers con- 
tinue to thrive. So long as the pie keeps getting bigger, there is room 
for both models. 


The challenge for all but the largest ISPs is differentiation. Smaller 
providers must give consumers some reason to choose them over AOL, especial- 
ly as transmission becomes commoditized. These smaller companies are there- 
fore exploring content bundling, Internet telephony, content filtering, spam 
blocking and other value-added services to set them apart. 


Smaller ISPs can also take advantage of the layered architecture of the 
Internet to avail themselves of many of the benefits that larger ISPs enjoy. 
An ISP is comprised of three components: Network operation, customer inter- 
action and application servers. Only the first of these requires local 
infrastructure. Conventional wisdom has held for some time that Internet 
access was fundamentally a network business, and would therefore come to be 
controlled by telcos. On the other hand, companies such as Mindspring and 
Earthlink have demonstrated that an ISP need not own facilities to be suc- 
cessful. Many of the core ISP functions (such as mail and Web hosting) are 
now available from Web-based providers such as GeoCities and Microsoft’s 
Hotmail. Roaming services allow local ISPs to offer global access points to 
their customers as a value-added feature, without the need for additional 
infrastructure investment. 


Virtual infrastructures are emerging on top of physical networks (see 
Release 1.0, 12-95). The layered architecture of the Internet allows simi- 
lar services to be provided at different levels of the protocol stack. 
Internet telephony imitates a low-level pure transmission service over high- 
level protocols in order to leverage network economies and avoid regulatory 
constraints. Connectivity and applications need not be tied to the same 
provider; users can assemble “virtual ISPs” using piece-parts from many dif- 
ferent companies. Companies are also bundling Internet access with other 
product and services, such as PCs. With VPNs, users can securely connect 
to corporate intranets from remote sites, blurring the line between the pub- 
lic Internet and private networks. 


The emergence of virtual infrastructures has so far had little effect on the 
price of connectivity. Most ISP costs are tied to hardware and phone 

lines, which remain the same regardless of the additional services ISPs pro- 
vide. As we’ve seen throughout this issue, the many factors driving pricing 
and service quality don’t lend themselves to easy generalizations. 
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APPENDIX: THE ECONOMICS OF BANDWIDTH TODAY 


Virtually all Internet traffic today runs over local access and long-haul 
networks of telephone companies. (See Release 1.0, 12-95 and 11-93, for 
earlier discussions of the economics of connectivity.) A telephone call 
involves five distinct segments: Originating customer premises, originating 
local access, interoffice transport, terminating local access, and terminat- 
ing customer premises. In the US and most other countries, the phones and 
inside wiring that make up the customer premises segments belong to the cus- 
tomer. Local loop costs vary widely with density, but not with usage, 
because the entire loop is dedicated to a particular customer. Switch costs 
and some interoffice facilities, however, are shared across multiple cus- 
tomers, and therefore one person’s usage constrains the capacity available 
to others. 


Interexchange Carrier 


Switch 


Originating Local Terminating Local i 
Customer Exchange Carrier Exchange Carrier Customer 


Premises Premises 


Diagram 1 — Simplified telephone network architecture 


The Internet rides on top of this telephone infrastructure. To the telco, 
ISPs are simply customers that receive large numbers of calls, and that 
lease significant volumes of private line capacity from local and long-dis- 
tance companies. 


ISP networks are technically different from telephone networks. Traditional 
telecommunications networks employ circuit switching. In other words, the 
network holds open an end-to-end channel for the duration of each call. 
Digital circuit-switched networks employ time-division multiplexing to carry 
multiple calls over the same facilities; every call has a dedicated set of 
time slices. The Internet uses packet switching, routing packets independ- 
ently through multiple network paths. 


Money flows into ISPs in much the same way as it flows into local phone 

companies. Users pay a monthly fee for access. However, an Internet con- 
nection does not have a single end-point, and packets may be routed across 
multiple ISP networks to reach various destinations. Consequently, ISPs 

must interconnect with each other, but it makes little sense to talk about 
a terminating access charge in the Internet context. Backbone providers 

charge downstream ISPs to transport their traffic across the Internet, and 
these backbone providers have traditionally connected to each other on the 
basis of “peering” (no settlement payments). As discussed above (see page 
11), the question of who gets to peer has become increasingly contentious. 
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Diagram 2 — Simplified Internet architecture 


The bottleneck to faster (and cheaper) Internet access tends to be the local 
switch. On one side is the local loop, which is dedicated to a single user 
and is capable of delivering significant bandwidth through digital sub- 
scriber line technology. On the other side is the digital interoffice net- 
work, comprised of high-speed fiber-optic links. In the middle, however, is 
a big, complex, expensive circuit switch, optimized for voice calling pat- 
terns. 


Switch engineers, using decades of data, assume that an average call lasts 
about three minutes, and that an average retail user makes three calls in 
the busiest hour of the day. In most countries, customers are charged by 
the minute for local calls, so they tend to limit their phone and Internet 
usage (and if they do not, they generate additional revenue for the phone 
company). In the US, however, unmetered local service is the norm, and the 
average dial-up Internet session lasts more than 20 minutes. Local phone 
companies have had to spend hundreds of millions of dollars since 1996 to 
overcome switch congestion resulting from skyrocketing dial-up Internet 
usage. Most of this money has gone to purchase more conventional circuit- 
switching equipment, and thus has not addressed the core architectural prob- 
lem of a network designed for voice being used by data. 


Effects of regulation 


Telephone companies have long been regulated monopolies virtually everywhere 
in the world. This is changing. Most members of the European Community 
were required to open their telecommunications markets to competition on 
January 1, 1998. The US has already broken up AT&T and is working towards 
competition in local service. Last year, 69 countries signed a World Trade 
Organization agreement under which they committed to liberalize their commu- 
nications industries. Nonetheless, full and open competition remains the 
exception rather than the rule, and regulation continues to have a powerful 
effect on prices. 
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Traditional telecommunications regulation emphasizes categories. Regulators 
divide traffic geographically (local, long-distance, international), func- 
tionally (voice, data, video) and technologically (wireline, wireless, 
satellite). Different pricing structures apply to each of these categories, 
and in many cases different regulators have jurisdiction over different cat- 
egories. In the US, the FCC sets interstate rates, but state public utili- 
ty commissions set local rates. In the UK, the Office of Telecommunications 
(Oftel) regulates local and long-distance telephony, but has no jurisdiction 
over broadcasting. And so forth. 


Lost in this maze of categories is a simple reality — the justification for 
traditional categories has been eroding for some time. Many of the existing 
divisions are arbitrary; we have sharp divisions between local and long-dis- 
tance service in the US because of the corporate structure of pre-divesti- 
ture AT&T and the manner in which AT&T was broken up. Other aspects of the 
status quo, like pricing based on minutes of use, were tied to then-preva- 
lent network architectures. In a circuit-switched network, a switch can 
carry a limited number of calls but each call gets a full transmission 
path. Consequently, it makes sense to price by minute — longer calls tie 
up switch ports and require additional capacity to handle other calls. But 
packet switches work differently. The limiting factor on packet-switched 
networks is bandwidth, because no capacity is dedicated to an individual 
call. Packet-switched networks are “always-on,” and so the only relevant 
metric is the amount of data being sent. 


Regulation and monopoly have long inflated both costs and prices in communi- 
cations. In the US, “universal service” began as an AT&T marketing slogan 
to support the emergence of one dominant carrier, but evolved into a complex 
subsidy mechanism to reduce the price of local calls, especially in rural 
areas. Local phone companies in the US collect more than $20 billion per 
year in “access charges” for originating and terminating long-distance 
calls, even though their costs are only a fraction of that amount. These 
charges have been understood to subsidize local service in high-cost areas, 
but because the subsidies are implicit it becomes impossible to disentangle 
what is real and what is artificial. Similar mechanisms are used elsewhere 
in the world to subsidize governments and particular classes of calls. 
Internet telephony terrifies incumbents because it avoids many of these reg- 
ulatory games. 


For international calls, carriers impose “settlement charges” to terminate 
in their countries. These often-exorbitant charges subsidize either domes- 
tic phone service in those countries or other government projects. Service 
providers look to avoid these rates by routing traffic through third coun- 
tries, or by using callback services to reverse the direction of a call. 

One consequence, however, is that far more traffic now originates in the US 
(where outbound international calling is relatively cheap) than elsewhere. 
US carriers now pay roughly $5.5 billion in net annual settlement charges to 
foreign carriers, which has led the FCC to take aggressive steps to force 
countries to lower their rates. 


Similar patterns exist for Internet traffic: For example, OECD statistics 
show that 46.7 percent of Singapore’s global Internet connectivity is 
routed via Vancouver, Canada. Because the Internet originally developed in 
the US, and the most popular sites still reside there, service providers in 
other countries generally must pay to bring their traffic to exchange 
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points in the US. Telstra of Australia and KDD of Japan have sued the FCC 
for allowing this practice, arguing that international circuit costs should 
be split equally. The reality, however, is that the imbalance of traffic 
is due largely to the high price of bandwidth in many countries and the 
absence of local exchange points, both of which will change over time. 


Why competition matters... 


Incumbent telephone companies will not reduce their prices dramatically 
unless they face pressure from competitors. Despite the costs of switch 
congestion, the Internet has so far been a tremendous boon to local phone 
companies. Second phone lines, which are especially lucrative because they 
usually do not require additional infrastructure deployment, are now used in 
over 15 percent of US homes in large part due to modem usage. Corporate 
and ISP demand for dedicated data circuits has skyrocketed, representing a 
$5 to $10 billion annual business in the US. The cost of deploying a Tl 
circuit has dropped significantly thanks to high-speed digital subscriber 
line (HDSL) technology in the phone network, but prices have remained 
steady, leading to increased profit margins. Prices are even higher outside 
the US, where competition tends to be even more limited. 


The high prices for data circuits today create a barrier to deployment of 
broadband technologies. A phone company that can sell a 1.544 mbps Tl cir- 
cuit for $1,000 per month will think twice about offering similar capacity 
to users through asymmetric digital subscriber line (ADSL) at $50 or $100 
per month. Incumbents won’t cannibalize their own services until they 
become convinced that someone else will eat them first. 


...and why it’s hard 


The inefficiencies of the current local-access infrastructure have been 
clear for some time. Yet for all the talk of convergence and competition 
little has changed. Opening up or circumventing the local loop turns out 
to be very difficult. In the US, the Telecommunications Act of 1996 
requires incumbent local exchange carriers (ILECs) to provide competitors 
with access to their network at any reasonable point. For competitive local 
exchange carriers (CLECs) and independent ISPs to offer DSL services, they 
must gain access to “unbundled loops” between subscribers and the incum- 
bents’ central offices. Pricing these unbundled loops and provisioning 
them, however, turns out to be very tricky. A federal court in St. Louis 
struck down the FCC’s national pricing guidelines on the grounds that only 
state public utility commissions can set prices for local facilities, lead- 
ing to contentious battles in every state. 


About 20 small ISPs have recently advocated an alternate approach to the 
FCC. Rather than seeking unbundled access to incumbent networks, these ISPs 
have requested access to “unswitched clean copper” as a retail service. 
Common carriers such as the ILECs are required to provide services to cus- 
tomers upon reasonable request. The ISPs and their attorney, Chris Savage 
of Cole, Raywid and Braverman, argue that ILECs themselves could provide 
“data-ready” copper loops to ISPs and their customers, so that these end- 
users could attach DSL equipment and bypass local switches. The FCC has 
not yet acted on this request, but it provides an intriguing alternative 
worth trying. 
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A hopeful note? 


Slowly but surely local-access bandwidth bottlenecks are being overcome. 
Some 200,000 US subscribers use high-speed cable modems, and all of the Bell 
Operating Companies have announced plans to deploy ADSL within the next nine 
months. Similar services are being rolled out in Canada, Europe and Asia. 
Other broadband technologies including local multipoint distribution service 
wireless, access via satellite and data over electric power lines are in 
trials around the world. Even if the incumbents do not move quickly, oth- 
ers will find ways to respond to the pent-up demand for bandwidth. 

Whatever the future of communications looks like, we can be sure it will be 
different from the past. 
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(516) 293-3996; e-mail von98@pulver.com. 

**EDventure’s High-Tech Forum - Copenhagen, Denmark. 
Sponsored by EDventure Holdings. Call Daphne Kis, (212) 
924-8800; fax, (212) 924-0240; daphne@edventure.com; 
www.edventure.com. 

Human Resource Technology Conference and Exposition - 
Philadelphia, PA. Latest on knowledge management and cor- 
porate intranets. Sponsored by L.R.P. Call (800) 727- 
1227; fax, (703) 739-0489; lrpconf@lrp.com/ More informa- 
tion at www.lrp.com/Conferences/conferences.htm. 


* Events Esther plans to attend. 
# Events Kevin plans to attend. 


Lack of a symbol is no indication of lack of merit. 
The full, 
Please let us know about other events we should include. — Mari Katsunuma 
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